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Transform Table for Ink Sizing and Compression 
Cross-Reference to Related Applications 

[01] This application is related to U.S. Application No. 60/212,825, entitled "Methods for 
Classifying, Anchoring, and Transforming Ink Annotations", filed June 21, 2000, to U.S. 
Application No. 09/750,288, entitled "Classifying, Anchoring, and Transforming Ink", 

filed December 29, 2000, to U.S. Application No. (BW 03797.00132), 

entitled "Serial Storage of Ink and Its Properties", filed May 11, 2001, and to U.S. 

Application No. (BW 03797.00131), filed June 1, 2001, entitled "Information 

Storage Using Tables and Scope Indices", each of whose contents is expressly 
incorporated herein by reference as to their entireties. 

Field of the Invention 

[02] Aspects of the present invention are directed generally to apparatus and methods for 
controlling a graphical user interface (GUI). More particularly, aspects of the present 
invention relate to capturing and/or storing electronic ink in an efficient manner. 

Background of the Invention 

[03] Typical computer systems, especially computer systems using graphical user interface 
(GUI) systems such as Microsoft WINDOWS, are optimized for accepting user input 
from one or more discrete input devices such as a keyboard for entering text, and a 
pointing device such as a mouse with one or more buttons for driving the user interface. 
The ubiquitous keyboard and mouse interface provides for fast creation and modification 
of documents, spreadsheets, database fields, drawings, photos and the like. However, 
there is a significant gap in the flexibility provided by the keyboard and mouse interface 
as compared with the non-computer (i.e., standard) pen and paper. With the standard pen 
and paper, a user edits a document, writes notes in a margin, and draws pictures and other 
shapes and the like. In some instances, a user may prefer to use a pen to mark-up a 
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document rather than review the document on-screen because of the ability to freely 
make notes outside of the confines of the keyboard and mouse interface. 

[04] Some computer systems permit a user to draw on a screen. For example, the Microsoft 
READER application permits one to add electronic ink (also referred to herein as "ink") 
to a document. The system stores the ink and provides it to a user when requested. Other 
applications (for example, drawing applications as known in the art are associated with 
the Palm 3.x and 4.x and PocketPC operating systems) permit the capture and storage of 
drawings. These drawings include other properties associated with the ink strokes used to 
make up the drawings. For instance, line width and color may be stored with the ink. One 
goal of these systems is to replicate the look and feel of physical ink being applied to a 
piece of paper. However, physical ink on paper may have significant amounts of 
information not captured by the electronic collection of a coordinates and connecting line 
segments. Some of this information may include the thickness of the pen tip used (as seen 
through the width of the physical ink), the shape of the pen tip, the speed at which the ink 
was deposited, and the like. 

[05] Another problem has arisen in the storage of ink. The fact that ink may be created from 
number of different platforms (for example, from interactions between a stylus and a 
digitizer, from a mouse and the like) yields a variety of ink sizes (based at least in part on 
the resolution and sampling frequency of the digitizer or ink input system). To reduce the 
size of created ink with its variety of data points, one technique includes normalizing the 
size of the ink from the coordinate system in which it has been captured (referred to 
herein as the ink's native coordinates) to a different coordinate system. This 
normalization may have devastating effects on the ability of a handwriting recognition 
system to recognize and appropriately treat ink. For instance, Figure 12A shows ink with 
30 data points in its native coordinates (labeled A-AD). A handwriting recognition 
system may recognize the ink to spell out the word "ball." The data points may be 
represented by the following table: 
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[06] However, the device or system that captured the ink of Figure 12 A may need to decrease 
the size of the captured ink or may need to scale down the ink so as to be used properly 
by an application. An example of this decrease in size or scaling may be to divide all data 
points by 4. In a binary representation, this means that the least significant bit and the 
second least significant bit are eliminated. One may, for example, use the following table 
to show a binary division by 4. 
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[07] Application of this process results in the quantization of the ink. Other quantizations or 
roundings are possible. 

[08] The original set of coordinates when divided by 4 is as follows: 
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[09] Figure 12 B shows the resulting sets of data points. Here, a handwriting recognition 
system would likely interpret the ink of Figure 12 B as "lill" or "lell" or the like, far from 
the original ink meaning of "ball." It is noted that any loops in the rendered ink are due 
primarily to curve fitting algorithms in the present rendition of Figure 12B. It is equally 
as likely that a tighter fitting curve-fitting algorithm would reduce the loops in Figure 12 
B, thus making the ink even more meaningless and/or less recognizable. In short, 
attempting to work from transformed ink may yield poor results. 

[10] A further problem is shown in Figures 13. Ink captured on one environment may be 
distorted when displayed, rendered, or printed, in another environment. In general, tablets 
capture in a first coordinate system and covert the ink to a second coordinate system. This 
process is generally handled by the capturing system (for example, a digitizer tablet) 
itself. An application using the ink received from the tablet only sees the ink in relation to 
the second coordinate system. If the application (for example, a word processing 
program) receives ink from a number of different tablets, the possibility exists that the 
second coordinate systems may be incompatible. Figure 13 shows a possible 
incompatibility among captured ink. Specifically, Figure 13 includes a first tablet 1301 
with a capture resolution of 800x400. Other tablets 1302, 1303, and 1304 are shown with 
different resolutions of 200x200, 400x400 and 1000x1000, respectively. In each tablet 
context, the word "hi" was written. Despite the standard size of the word "hi" in all tablet 
contexts, the outputs of each tablet results in the ink having a different size. The 
comparative sizes of the ink from the tablets 1301-1304 are shown in output 1305. Here, 
the transformation that was applied to each ink is as follows: 
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1) The output of tablet 1301 was divided by 2; 

2) The output of tablet 1302 was multiplied by 2; 

3) The output of tablet 1303 was not modified (or modified by a factor of 1); 
and, 

4) The output of tablet 1304 was not modified (or modified by a factor of 1). 

[11] The ink from tablet 1304 has kept the same coordinate system yet the ink is so large that 
it dwarfs the other ink in output 1305, despite the fact that the ink of tablet 1304 was 
originally created with the same size drawing on the tablet 1304. 

[12] Yet another problem with handling ink occurs in compression of the ink after scaling. 
Figure 14A shows the statistical distribution of sample ink before scaling (0, 1,-1,2, -2, 
3, -3, ...). Figure 14B shows the ink when the ink has been scaled by multiplying the ink 
coordinates by a factor of 2 (0, 2, -2, 4, -4, 6, -6, ...). Here, the information contained 
within the ink of Figures 14A and 14B has not significantly changed. However, 
compressing the ink has become more difficult as larger numbers are used to represent 
the ink coordinates. Figure 14C shows a more complex example of scaling ink. Here, the 
original ink of 14A was multiplied by a factor of 1.5. Alternatively, the ink of Figure 14B 
was multiplied by a factor of .75. The resulting distribution of ink has become more 
complex as seen by the side lobes surrounding each primary peak (0, 1.5, -1.5, 3.0, -3.0, 
4.5, -4.5, ...). The information contained within the ink has not significantly changed. 
However, the data used to convey the information contained within the ink has become 
more complex. In one example, the complexity may have increased due to the increase in 
distribution of the second derivatives of coordinate values (also referred to as a delta- 
delta distribution). Compression of the ink has become more complex as the number of 
different peaks of information has increased. In short, scaling of the ink may have 
detrimental effects on the compressibiUty of the ink. 

[13] Accordingly, an improved system is needed for storing information with its associated 
properties that minimize the size of the data used to represent the desired information. 
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Summary of the Invention 

[14] The present invention provides a flexible and efficient system, method, and data structure 
for receiving, storing, and rendering electronic ink. 

[15] Aspects of the present invention are directed to an improved system, method and data 
structure for storing ink in a first coordinate system and a mapping to a second coordinate 
system. In some embodiments, the ink coordinates and the mapping (also referred to as a 
transform table) are stored together in an ink object. In further embodiments, the ink 
coordinates and mapping are stored in an ink object in a serialized format so as to 
minimize the size of the ink object. In yet other embodiments, the mappings may be 
combined to minimize degradation or loss of resolution of the ink (caused in some 
instances by rounding errors). Finally, some embodiments relate to defining and using a 
virtual ink space as the second coordinate system. 

[16] These and other features and aspects of the invention will be apparent upon consideration 
of the following detailed description of the preferred embodiments. 

Brief Description of the Drawings 

[17] The foregoing summary of the invention, as well as the following detailed description of 
preferred embodiments, is better understood when read in conjunction with the 
accompanying drawings, which are included by way of example, and not by way of 
limitation with regard to the claimed invention. 

[18] Figure 1 shows a schematic diagram of a general-purpose digital computing environment 
that can be used to implement various aspects of the invention. 

[19] Figure 2 shows a plan view of a tablet computer and stylus that can be used in accordance 
with various aspects of the present invention. 

[20] Figure 3 shows an example of a stroke or strokes of ink having points and properties in 
accordance with the present invention. 
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[21] Figure 4 shows ink from various platforms transformed in accordance with embodiments 
of the present invention. 

[22] Figure 5 shows a process for creating a data structure having a transform in accordance 
with embodiments of the present invention. 

[23] Figures 6 and 7 show processes for transforming ink in accordance with embodiments of 
the present invention. 

[24] Figure 8 shows a process for capturing ink in accordance with the present invention. 

[25] Figures 9 and 10 show various data structures for storing ink and associated properties in 
accordance with embodiments of the present invention. 

[26] Figure 11 shows a data structure storing a transform table and transform blocks in 
accordance with embodiments of the present invention. 

[27] Figures 12A and 12B show ink before and after resizing. 

[28] Figure 13 shows ink from various capturing environments. 

[29] Figures 14A-14C show statistical distributions of ink before and after resizing. 

Detailed Description of Preferred Embodiments 

[30] The following description is divided into sub-sections to assist the reader. The sub- 
sections include: characteristics and storage of ink; terms; general-purpose computer and 
associated hardware; an example of strokes of ink; transforming ink; data structures for 
storing ink and transform mappings; transform properties; encoding of values; and a 
sunnmarization of the storage of ink. 

[31] While described with respect to the storage of ink, it is appreciated that the storage 
structures defined herein may be applied to non-ink items as well. For the purposes of 
this disclosure, ink objects are used as an example. Other objects reflecting information 
having properties may equally be used as well but are omitted for simplicity. For 
example, the storage structures may be applied to text (with properties including bold, 
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font face, underline, margin settings and the like), graphics (with properties including 
modifications), non-modifiable information with subsequent modifications (for example, 
non-modifiable images with properties comprising subsequent comments), displayed 
information, and the like. The transform aspects described herein may be applied to the 
various objects and are considered within the scope of the present invention. 

Characteristics and Storage of Ink 

[32] The present invention supports the storage of electronic ink and/or properties or other 
data associated with electronic ink. Ink as used herein refers to electronic ink. Ink refers 
to a sequence of strokes, where each stroke is comprised of a sequence of points. The 
points may be represented using a variety of known techniques including Cartesian 
coordinates (X, Y), polar coordinates (r, 0), and other techniques as known in the art. 

[33] As known to users who use ink pens, physical ink (the kind laid down^ on paper using a 
pen with an ink reservoir) may convey more information than a series of coordinates 
connected by line segments. For example, physical ink can reflect pen pressure (by the 
thickness of the ink), pen angle (by the shape of the line or curve segments and the 
behavior of the ink around discreet points), and the speed of the nib of the pen (by the 
straightness, line width, and line width changes over the course of a line or curve). 

[34] To provide the look and feel of physical ink, the exemplary disclosed system and method 
store ink strokes and properties associated with the ink strokes to more fully render ink. 
In some embodiments, ink may be stored as a series or set of strokes and a series or set of 
properties. In other embodiments, ink may be stored with a complex series of properties 
in which the properties have properties of their own. Properties of the ink may include, 
for example, color, width, pressure between the stylus and tablet, and angle between the 
stylus and tablet, and pen shape and the like. While these properties may suffice for many 
applications, embodiments of the present invention provide for the extensible storage of 
custom properties (and other data) generated by applications. All strokes and values may 
be stored directly with excess information. However, alternative embodiments reflect 
considerations that eliminate excess information when possible or practicable. 
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[35] One of these properties may include a mapping of a coordinate system to another. For 
example, ink may be stored in its original coordinate system. The ink may be associated 
with a transform mapping from the original coordinate system to another coordinate 
system. In further embodiments, the mapping may be altered to reflect the application of 
the ink to further coordinate systems. 

[36] As a part of the ink object, a transform mapping may be stored to represent the transform 
of the ink in its native coordinates to a virtual ink space. At a later time, the mapping may 
be used to transform the ink. In further embodiments, a second transform that maps 
coordinates from the virtual ink space to an output (for example, a printer, display, 
graphical file, handwriting recognition system, and the like) may be combined with the 
original mapping to the virtual ink space. The combination of the mappings may be 
applied to the ink in its native coordinates to directly transform the ink into a final 
coordinate system. It is appreciated that further combinations of mappings are realizable 
as well. 

Terms 

[37] Ink - A sequence or set of strokes with properties. A sequence of strokes may include 
strokes in an ordered form. The sequence may be ordered by the time captured or by 
where the strokes appear on a page. Other orders are possible. A set of strokes may 
includes sequences of strokes or unordered strokes or any combination thereof 

[38] Stream - A sequence of strokes that may or may not include properties that comprises a 
data structure. 

[39] Ink object - A data structure storing a stream with or without properties. 

[40] Stroke - A sequence or set of captured points. For example, when rendered, the sequence 
of points may be connected with lines. Alternatively, the stroke may be represented as a 
point and a vector in the direction of the next point. In short, a stroke is intended to 
encompass any representation of points or segments relating to ink, irrespective of the 
underlying representation of points and/or what connects the points. 
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[41] Point - Information defining a location in space. For example, the points may be defined 
relative to a capturing space (for example, points on a digitizer), a virtual ink space (the 
coordinates in a space into which captured ink is placed), and/or display space (the points 
or pixels of a display device). 

[42] Space - A context in which objects have meaning. For example, a digitizer space may 
include the capturing of ink (or other information). A rendering space may refer to the 
context of determining how information is to be rendered. A virtual ink space may be 
considered to exist between the digitizer space and the rendering space. 

[43] Virtual Ink Space - A framework to which all ink strokes relate. The framework may 
include a two- or three- (or more) dimensional shape. In one example, the framework 
may include a unit size square. In another example, the framework may include a defined 
rectangle, referred to as an ink space rectangle. While some ink strokes may extend 
outside of the framework, the framework may be used as a starting point for rendering 
purposes including dimensioning for a printer or a display or subsequent processing by, 
for example, a handwriting recognition system, or the like. In one aspect, the virtual ink 
space is a norm to which ink strokes may be spatially defined. 

[44] Ink Space Rectangle - a defined rectangle to which ink may be referenced. For example, 
when ink is captured, the mapping to the virtual ink space may include its relation to the 
ink space rectangle. 

[45] Transform Table and Transform Block - A mapping from one coordinate set to another 
coordinate set. The mapping may include equations, coefficients, from and to rectangles, 
or any combinations of these as well as others. The mapping may be provided by itself, in 
combination with an ink object, in combination with other mappings, or in combination 
with other mappings in an ink object. When used in combination with an ink object, the 
mapping may be specified in a transform block. When used with other transform blocks, 
the transform blocks may be combined into a transform table. 
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[46] Global Ink Properties - These are properties that apply to a stroke or set of strokes unless 
otherwise defined. For example, a selected ink color may be blue. By setting all strokes to 
blue, the default color of the strokes would be blue. 

[47] Local Ink Properties - These are properties that apply to a specific stroke (or data point or 
data points). For example, while a global ink property may be blue, a specific stroke may 
be set to red. Some local ink properties may be interpreted, in some cases, as global 
properties as they affect subsequently encountered strokes in an ink object. It is noted that 
properties may or may not be labeled as global or local. In some examples, the created 
data structure defines the scope of the properties. 

[48] Render - The process of determining how text, graphics, and/or ink is to be displayed, 
whether on a screen or printed, or output into another file format. 

General Purpose Computer 

[49] Figure 1 illustrates a schematic diagram of an exemplary conventional general-purpose 
digital computing environment that can be used to implement various aspects of the 
present invention. In Figure 1, a computer 100 includes a processing unit 110, a system 
memory 120, and a system bus 130 that couples various system components including the 
system memory to the processing unit 110. The system bus 130 may be any of several 
types of bus structures including a memory bus or memory controller, a peripheral bus, 
and a local bus using any of a variety of bus architectures. The system memory 120 
includes read only memory (ROM) 140 and random access memory (RAM) 150. 

[50] A basic input/output system 160 (BIOS), containing the basic routines that help to 
transfer information between elements within the computer 100, such as during start-up, 
is stored in the ROM 140. The computer 100 also includes a hard disk drive 170 for 
reading from and writing to a hard disk (not shown), a magnetic disk drive 180 for 
reading from or writing to a removable magnetic disk 190, and an optical disk drive 191 
for reading from or writing to a removable optical disk 192 such as a CD ROM or other 
optical media. The hard disk drive 170, magnetic disk drive 180, and optical disk drive 
191 are connected to the system bus 130 by a hard disk drive interface 192, a magnetic 
disk drive interface 193, and an optical disk drive interface 194, respectively. The drives 
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and their associated computer-readable media provide nonvolatile storage of computer 
readable instructions, data structures, program modules and other data for the personal 
computer 100. It will be appreciated by those skilled in the art that other types of 
computer readable media that can store data that is accessible by a computer, such as 
magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random 
access memories (RAMs), read only memories (ROMs), and the like, may also be used in 
the example operating environment. 

[51] A number of program modules can be stored on the hard disk drive 170, magnetic disk 
190, optical disk 192, ROM 140 or RAM 150, including an operating system 195, one or 
more application programs 196, other program modules 197, and program data 198. A 
user can enter commands and information into the computer 100 through input devices 
such as a keyboard 101 and pointing device 102. Other input devices (not shown) may 
include a microphone, joystick, game pad, satellite dish, scanner or the like. These and 
other input devices are often connected to the processing unit 110 through a serial port 
interface 106 that is coupled to the system bus, but may be connected by other interfaces, 
such as a parallel port, game port or a universal serial bus (USB). Further still, these 
devices may be coupled directly to the system bus 130 via an appropriate interface (not 
shown). A monitor 107 or other type of display device is also connected to the system 
bus 130 via an interface, such as a video adapter 108. In addition to the monitor, personal 
computers typically include other peripheral output devices (not shown), such as speakers 
and printers. In a preferred embodiment, a pen digitizer 165 and accompanying pen or 
stylus 166 are provided in order to digitally capture freehand input. Although a direct 
connection between the pen digitizer 165 and the processing unit 110 is shown, in 
practice, the pen digitizer 165 may be coupled to the processing unit 110 via a serial port, 
parallel port or other interface and the system bus 130 as known in the art. Furthermore, 
although the digitizer 165 is shown apart from the monitor 107, it is preferred that the 
usable input area of the digitizer 165 be co-extensive with the display area of the monitor 
107. Further still, the digitizer 165 may be integrated in the monitor 107, or may exist as 
a separate device overlaying or otherwise appended to the monitor 107. 
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The computer 100 can operate in a networked environment using logical connections to 
one or more remote computers, such as a remote computer 109. The remote computer 
109 can be a server, a router, a netw^ork PC, a peer device or other common network 
node, and typically includes many or all of the elements described above relative to the 
computer 100, although only a memory storage device 111 has been illustrated in Figure 
1. The logical connections depicted in Figure 1 include a local area network (LAN) 112 
and a wide area network (WAN) 113. Such networking environments are commonplace 
in offices, enterprise-wide computer networks, intranets and the Internet. 

When used in a LAN networking environment, the computer 100 is connected to the 
local network 112 through a network interface or adapter 114. When used in a WAN 
networking environment, the personal computer 100 typically includes a modem 115 or 
other means for establishing a communications over the wide area network 113, such as 
the Internet. The modem 115, which may be internal or external, is connected to the 
system bus 130 via the serial port interface 106. In a networked environment, program 
modules depicted relative to the personal computer 100, or portions thereof, may be 
stored in the remote memory storage device. 

It will be appreciated that the network connections shown are exemplary and other 
techniques for establishing a communications link between the computers can be used. 
The existence of any of various well-known protocols such as TCP/IF, Ethernet, FTP, 
HTTP and the like is presumed, and the system can be operated in a client-server 
configuration to permit a user to retrieve web pages from a web-based server. Any of 
various conventional web browsers can be used to display and manipulate data on web 
pages. 

Figure 2 illustrates an exemplary tablet PC 201 that can be used in accordance with 
various aspects of the present invention. Any or all of the features, subsystems, and 
functions in the system of Figure 1 can be included in the computer of Figure 2. Tablet 
PC 201 includes a large display surface 202, e.g., a digitizing flat panel display, 
preferably, a liquid crystal display (LCD) screen, on which a plurality of windows 203 is 
displayed. Using stylus 204, a user can select, highlight, and/or write on the digitizing 



14 



Banner & Witcoff, Ltd. 
03797.00133/171945.1 

display surface 202. Examples of suitable digitizing display surfaces 202 include 
electromagnetic pen digitizers, such as Mutoh or Wacom pen digitizers. Other types of 
pen digitizers, e.g., optical digitizers, may also be used. Tablet PC 201 interprets gestures 
made using stylus 204 in order to manipulate data, enter text, create drawings, and/or 
execute conventional computer application tasks such as spreadsheets, word processing 
programs, and the like. 

[56] The stylus 204 may be equipped with one or more buttons or other features to augment its 
selection capabiUties. In one embodiment, the stylus 204 could be implemented as a 
"pencil" or "pen", in which one end constitutes a writing portion and the other end 
constitutes an "eraser" end, and which, when moved across the display, indicates portions 
of the display are to be erased. Other types of input devices, such as a mouse, trackball, 
or the like could be used. Additionally, a user's own finger could be the stylus 204 and 
vO used for selecting or indicating portions of the displayed image on a touch-sensitive or 

.|3 proximity-sensitive display. Consequently, the term "user input device", as used herein, 

iZ is intended to have a broad definition and encompasses many variations on well-known 

^^ input devices such as stylus 204. Region 205 shows a feedback region or contact region 

r permitting the user to determine where the stylus 204 as contacted the display surface 

S 202. 

Si [57] In various embodiments, the system provides an ink platform as a set of COM 
r: (component object model) services that an application can use to capture, manipulate, and 

store ink. One service enables an application to read and write ink using the disclosed 
representations of ink. The ink platform may also include a mark-up language including a 
language like the extensible markup language (XML). Further, the system may use 
DCOM as another implementation. 

An Example of Strokes of Ink 

[58] An exemplary ink object is shown in Figure 3. The ink object starts at point 301 where a 
pen down action occurred. The pen down action may be stylus 204 contacting the display 
surface 202, the click of a mouse button, the operation of a button on a trackball or 
joystick, or the like. The user controls an input (such as stylus 204) device and the 
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resulting stroke continues through points 302-316. At point 316, a pen up action 
occurred. The pen up action may be the lifting of the stylus 204 off the display surface 
204, releasing or another operation of a mouse button, or the operation of the button (or 
other buttons) on the trackball or joystick or the like. Here, a pen up action and a pen 
down action are known in the pen digitizing art, 

[59] From points 301 through 308, the width of the stroke has a first value. At point 308, the 
width of the stroke changes to a second value. This may have been because the user 
increased the pressure between the stylus 204 tip and the display surface 202, because the 
angle between the stylus 204 and the tablet changed, because the stylus 204 was rotated 
and projected a different cross section of the stylus 204' s nib, or the like. The stroke then 
continues through point 316 with the second stroke width. In an alternate embodiment, a 
user started the stroke with a first line width and selected a different line width at point 
308 to complete the stroke. In a further embodiment, two strokes may form the ink object 
as shown in Figure 3. For example, a first stroke may include points 301-308 and a 
second stroke may include points 308-316. 

[60] In a further embodiment, the ink of Figure 3 may be represented as four or more strokes. 
Here, the stroke or strokes from points 301 to 306 may be blue (represented by group 
317) with the first stroke width, the stroke or strokes from points 306 to 308 may be 
green (group 318) with the first stroke width, the stroke or strokes from points 308 to 309 
may be green (also as part of group 318) with the second stroke width, and strokes or 
strokes from points 309 to 316 may be red (group 319) with the second stroke width. 

[61] Next, the ink object may be stored (or transmitted or displayed or the like). The ink object 
stroke may be represented as a single stroke with varying line widths and colors. 
Alternatively, the ink object may be stored as a variety of strokes having a few data 
points in which each stroke has its own set of properties. Third, the ink object may be 
stored as short strokes between points. In short, the ink object may represent a stroke in a 
variety of forms. 
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Transforming Ink 

[62] Ink objects may be created with a number of different input sources. The input sources 
may include a variety of tablets with different tablet resolutions. The different tablet 
resolutions may result in ink drawn on a screen be rendered incorrectly when the created 
ink is ported to and displayed on a tablet having a different resolution or screen size. To 
adjust the ink to a common size, a set of transform properties (or transforms) may be 
stored to indicate how the ink object is to be adjusted to a conmion space. 

[63] As described above, ink may be captured in a variety of ways. The ink may be 
transformed larger or smaller, rotated, translated, warped, and the like. Along these lines, 
ink may need to be transformed from its native coordinate system to a different 
coordinate system. For example, a tablet may capture ink in a first resolution, transform 
the ink to a second resolution, and output the ink in the second resolution to an 
application. 

[64] Figure 4 shows ink from various platforms in their respective native coordinates 
transformed to a second coordinate system relating to a virtual ink space. Table 401 has a 
resolution of 800x400 points for capturing ink. Tablets 402, 403, and 404 have 
resolutions of 1000x1000, 200x200, and 400x400, respectively. Ink captured on these 
tablets includes a property that relates them to the virtual ink space. In this example, the 
virtual ink space is shown as a 400x400 resolution rectangle 405. The virtual ink space is 
not limiting. Rather, ink may extend beyond the sides of rectangle 405 specified above. 
Along this line, the virtual ink space may be used as an initial framework to permit an 
application to appreciate an intended size of the ink and relationships between ink. 

[65] Blocks 406, 407, 407, and 409 represent transformation from the initial coordinate sets of 
tablets 401, 403, 404, and 402, respectively, to the virtual ink space. The ratio for 
mapping tablet 401 to the virtual ink space by the transform 406 may be 1:1 for both the 
X and Y directions with the origin of tablet 401 placed at the at point A in virtual ink 
space 405. Note that points in the X direction from the tablet 401 exceed the length of the 
rectangle 405. The ratio for mapping tablet 403 to the virtual ink space by the transform 
407 may be 1:1 for both the X and Y directions with the origin of tablet 403 placed at the 
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at point B in rectangle 405. The ratio for mapping tablet 404 to the virtual ink space by 
the transform 408 may be 1 : 1 for both the X and Y directions with the origin of tablet 404 
placed at the at point A in rectangle 405. The ratio for mapping tablet 402 to the virtual 
ink space by the transform 409 may be 1000 : 400 (or 1000/400 : 1 or 5 : 2 or 5/2 : 1) for 
both the X and Y directions with the origin of tablet 402 placed at the at point B in 
rectangle 405. 

[66] Figure 5 shows an exemplary process for creating a data structure having a transform. 
First, in step 501, stroke information is received. This information may come from a 
display surface 202 or any other source that is capable of generating strokes. Next, in step 
502, the system 201 parses the stroke information 501. The system may parse a number 
of items including, for example, pressure information, stylus 204 tilt information, 
position information and the like. The parsed information may be temporarily stored in 
step 504 (as shown as an option by the dotted box of 504) and properties added in step 
505 (for example, global and/or local properties). 

[67] Tablet information 502 may be obtained and parsed in step 503. Alternatively, tablet 
information 502 may be obtained and added as a transform mapping property in step 505. 
Next, in step 507, the data structure is stored. Here, the storage step 507 is considered 
optional as other things may occur to the created ink object including, but not limited to, 
the inclusion of additional transformations in the ink object. Altematively, the data 
structure may be forwarded to another device, displayed, further manipulated, and the 

like. Co-pending U.S. Serial Nos. (BW 03797.00132) and U.S. Serial No. (BW 

03797.00131) disclose various tagged structures, which are incorporated by reference. 

[68] Figures 6 and 7 show processes for transforming ink. Figure 6 shows multiple transforms 
of original ink 601 resulting in ink C 607. Original ink 601 its original coordinate 601 is 
transformed (with a first transform 1 602) to ink A 603. Ink A 603 is transformed (with a 
second transform 2 604) to ink B 605. Finally, ink B 605 is transformed with transform 3 
606 to ink C 607. 

[69] Figure 7 shows an alternative process for transforming original ink 701 into ink A 703, 
ink B, 705, and ink C 707. As with Figure 6, original ink 701 may be transformed using 
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transform 1 in step 702 to form ink A 703. In step 704, however, transform 1 and 
transform 2 may be combined and applied to original ink 701 to form ink B 705. Further, 
in step 706, transform 1 may be combined with transform 2 and transform 3 and then 
applied to the original ink 701 to form ink C 707. At least one advantage of combining 
transforms prior to application to ink is that transforming the ink once minimizes 
degradation of the ink and/or minimizes the complexity of the ink. 

[70] Figure 8 show a process for creating and rendering ink. In step 801, the system receives 
an event indicating that the ink is about to be created. Here, this event is referred to as a 
"pen down" event. In step 802, the system creates an empty ink object for storing the 
created ink. In step 803, the system defines a mapping from the capturing device to a 
virtual ink space. Here, the mapping may include a set of equations, a listing of 
coordinates, and the like. Further, the definition of the tablet space to the virtual ink space 
may be the specification of the tablet and the specification of the virtual ink space. An 
example of combining transforms is shown in relation to equations 8-15 below. 

[71] Next, in step 804, a stroke is added to the ink object. In an alternative embodiment, 
strokes may be added as a group to the ink object. In step 805, the system determines 
whether an event has occurred that signals the end of the ink stroke. This event may be a 
"pen up" event or any other event that may indicate that no more strokes are to be added 
to the ink object. In an alternative embodiment, the ink object may have strokes added 
even after the pen up event or alternative event. 

[72] In step 806, a transform mapping may be applied to the strokes. For example, if a tablet 
has a high resolution and the ink object is not to store coordinates in such a high 
resolution, an initial transform may be applied to the stroke or strokes in the ink object in 
step 806. 

[73] In step 807, the ink object is finished. The finished ink object may be stored or otherwise 
handled. For example, the stroke or strokes may be transformed in step 808. Also, the ink 
object may be rendered in step 809. The steps 806, 808, and 809 are optional in this 
process as shown by their dotted boxes. Further, the definition 803 of the transform or 
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transforms to be applied to ink strokes may be determined after the ink strokes have been 
added to the ink object. 

Data Structures for Storing Ink and Transform Mappings 

[74] An ink object is one of a variety of structures for storing ink. Figure 9 shows an 
embodiment of a data structure used for storing ink. Other data structures are shown in 

related U.S. Application Serial Nos. (BW 03797.00132) and (BW 

03797.00131), which are expressly incorporated by reference. Ink object 900 with an ink 
object identifier 901 contains a property table 903 and five strokes (907, 909, 910, 912, 
and 913). The property table 903 (with property identifier or tag 902) includes property 
blocks 904, 905, and 906). The ink object 900 also contains two indices (an index 908 to 
property block 2 905 and an index 911 to property block 1 904). The two indices are 
located between strokes 1 907 and 2 909 and between strokes 3 910 and 4 912, 
respectively. Stroke 1 907 does not have a preceding index. In one example, stroke 1 907 
may have properties specified by a default property block (not shown). In another 
example, stroke 1 907 may have an implicit index to the first property block (here, 
property block 1 904). In a third example, stroke 1 907 may not appear (rather, pushed 
down in the data structure to appear after at least one index) as shown by the dotted box 
of stroke 1 907). 

[75] When more than one property block occurs, it may be stored with other property blocks 
in a property table. The specific property blocks may be specified by an index referencing 
a specific property block or a default property block. Strokes 3 910 and 5 913 do not have 
immediately preceding indices. In one example, this may indicate that strokes 3 910 and 
5 913 are to have properties specified by the defauh property block (not shows). In an 
alternate example, the strokes 3 910 and 5 913 use the most recent preceding index. 
Stroke 3 910 would use index 808. Stroke 5 913 would use index 911. Eliminating the 
recitation of an index for strokes 3 910 and 5 913 helps reduce the size of the ink object 
by the space that would have been consumed by separate indices for strokes 3 810 and 5 
813. 



20 



Banner & Witcoff, Ltd. 
03797.00133/171945.1 

Figure 10 shows an embodiment of a property table 1000. The table 1000 includes a 
property table tag or identifier 1001. Following identifier 1001 is the size or count of the 
number of table entries 1002. The table 1000 contains property block 1004 with identifier 
1003 and size or count information 1005 followed by the data for the property block 
1006. Also contained in the table 1000 is another property block 1008 with tag or 
identifier 1007 with size or count information 1009 followed by the data for the property 
block 1010. 

Here, the identifier 1001 is generally referred to as a tag (or TAG or Tag). A "tagged" 
structure, as shown in Figure 10, begins with an identifying "tag" followed by a "size 
field" followed by data. The "tag" identifies the contents of the data while the "size field" 
identifies, for example, the size of the data in bytes (or bits and the like). The tag may be 
either a predefined tag or an application-specific custom tag. In alternate embodiments, 
the size of the tagged field may appear prior to the tag itself. 

The structure as shown in Figure 10 may also include a count of the number of objects, 
tags, properties, strokes, and the like contained within it. In this regard, the "count" 
identifier may be used in place of the "size" identifier. If one specifies the size of the rest 
of the data of the tag, a system may then quickly skip over the rest of the data of the tag if 
desired. On the other hand, if the count of the number of objects (or properties or the like) 
was specified, the physical size of the count would likely be smaller than the physical 
size of the remaining data. In this regard, the ink object would be smaller if a count of 
remaining objects (or the like) was used rather than a size of the remaining data. 
However, to skip over the remaining part of an ink object or property, one may need to 
enumerate all of the sub-objects (or sub-tags or sub-properties) contained within the 
object or tag. To enumerate these sub-parts, a system may need to perform additional 
calculations to obtain the number of sub-parts. Further, a system may need to perform 
additional steps on the skipping operation (for example, advancing past the present object 
or tag or property) by counting the sub-parts in a count-based data structure, rather than 
advancing to a new position as used in a size-based system. 
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[79] One benefit of placing the size after the tag itself is that applications that do not recognize 
the tag may know, by reading the next portion of information (the size block 1002, 1005, 
1009), the length of data needed to skip over to arrive at the next tag or end of the ink 
data structure. 

[80] As to the specification of size of following data or the count of items in following data, it 
is appreciated that one may use either of the two ways of specifying information. For 
simplicity, the following description includes the use of size information. In some 
instances, count information is also shown. However, where only size information is 
shown, count information may readily be specified in place of or in addition to the size 
information. 

[81] As applied to ink, the tagged data structure of Figure 10 may be enhanced in a number of 
ways. In some embodiments, ink strokes may be defined to come in order. In other 
embodiments, global properties (or properties that may affect all subsequent ink strokes) 
are provided at a known location for the system. For example, all global properties may 
be specified at the beginning of the ink object. On the other hand, all global properties 
may be specified at the end of the ink object. One advantage of putting the global 
properties at the beginning of the ink object is that a system would already know how to 
handle a stroke that referenced global properties once it encounters the stroke (as it would 
have already encountered the global properties section of the ink object). In yet more 
embodiments, custom properties may be defined through various tables. The use at least 
one of these or other enhancements permit properties to be used throughout the ink object 
and permit a more efficient storage of ink information. 

[82] The "tag" describing the "data" indicates whether the "data" contains tagged structures or 
even how many tagged structures. For example, custom properties are considered opaque 
to the present system since they are application-defined and thus only contain data. 
However, a stroke may have one or more stroke properties, which are also represented as 
tagged structures. 

[83] Ink properties may be defined to minimize or eliminate redundant information. In simple 
streams, there may be no ink properties. However, if there are ink properties, then it is 
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preferable that they appear before any strokes so that they may apply to all strokes when 
needed. It is appreciated that the properties may be placed after the strokes as well. A 
number of global properties may be used. More or less global properties may be defined 
and used as needed. 

[84] With respect to ink objects, common properties appear. The properties may be grouped 
into transformation properties, drawing attributes properties, metrics properties, and 
stroke description properties. Other properties may be defined and used as well. Also, not 
all (if any) of the specific above-identified properties are required to practice the 
invention. Sample implementations of the above conmion properties are shown. Other 

properties are addressed in greater detail in co-pending U.S. Serial No. (BW 

03797.00132) and U.S. Serial No. (BW 03797.00131), which are incorporated 

^ by reference. 

iB Transform Properties 

1^ [85] Figure 1 1 shows a transform table 1 100 as including transform blocks 1 104 and 1 108 that 

U represent the various transform properties that may be used by an ink stream. Each 

^ transform block 1104, 1108 may define a unique transform that is applied to the ink 

13 points before they are rendered or used. These blocks 1104, 1108 may apply to one or 

II J more strokes and are placed in table 1 100 so that they are not repeated in each stroke. 

O [86] One may use the identifier of TAG_TRANSFORM_TABLE as the identifier 1101 to 
identify the transform table 1100. The size or count of the table may follow 1102. The 
size of the table may be is equal to the sum of the sizes of all transform blocks. The count 
may be equal to the number of blocks 1104, 1108 in the transform table 1100. Each 
transform block 1104, 1108 may include a transform block identifier 1103, 1107, a size 
or count field 1105, 1109 and a data field 1106, 1110. 

[87] To transform a point (X, Y) in two dimensions to a new point (X', Y'), one may use the 
following two equations: 

X' =xMlH-yM12 + Dx (1) 



23 



Banner & Witcoff, Ltd. 
03797.00133/171945.1 

Y' = xM21 + yM22 + Dy (2) 

The two equations provide a general solution in two dimensions for both rotation and 
translation of a point or points as based on the transformation matrix of six coefficients, 
namely Mil, M12, M21, M22, and Dx and Dy (Mil, M12, M21, and M22 being the 
rotational and scalar coefficients and Dx and Dy being the translational coefficients. A 
matrix of the six coefficients may take the form of: 

Mil M12 Dx 
Mil Mil Dy 

A general equation for three dimensions may be given by: 

X' =xMll +yM12 + zM13+Dx (3) 

Y' = xM21 + yM22 + zM23 + Dy (4) 

Z' = xM3 1 + yM32 + zM33 + Dz (5) 
A specific solution without rotation for two dimensions may be given by: 

X'=xMll+Dx (6) 

Y'=yM22 + Dy (7) 

The following description relates to transforms in two dimensions. It is appreciated that 
any n-dimension transforms may be used. 

The specification of the six coefficients for the transform properties of equations 1 and 2 
may be preferable to only specifying the four coefficients of equations 6 and 7 due to the 
fact that tablets (and/or capture systems may orient captured ink differently, for example, 
the origin being at the top left comer of a screen as opposed to the bottom left of a 
screen). On the other hand, specifying only four coefficients takes less space. 

In one example, if the size of the four or six coefficients is constant (as opposed to the 
actual values of the coefficients), the size or count fields 1105, 1109 may be eliminated. 
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Further, a transform table containing only one transform block is a special case. As with 
other properties in which there is only one block, the table tag 1101 and size/count 1102 
for the table may be omitted and the entire table may be replaced by a single transform 
block. 

In the simplest case, where the capturing environment and the rendering environment 
have no transformations between them, as well as in cases where scaling and transforms 
have been applied outside of a native capture environment, no transform tables may be 
created. 

To assist in the transformation to a common coordinate space, an ink space rectangle that 
defines a virtual coordinate space (or virtual ink space) for receiving ink may be 
specified. One tag that may be used includes TAG_INK_SPACE„RECT, which identifies 
the ink space rectangle when present in the stream. This rectangle may be, for example, 
rectangle 405 from Figure 4. The entry for the rectangle may have a size field. On the 
other hand, the entry for the rectangle may not need a size field the rectangle has a fixed 
number of elements of, for example, four signed numbers. These four numbers represent 
the left, top, right, and bottom of the ink space. The ink space rectangle defines the virtual 
coordinate space for the ink. An application may use this rectangle to determine what 
area of the ink to either display or print. The ink space rectangle may be referred to as 
defining a virtual sheet of paper that the ink is drawn on. This does not mean that ink may 
not appear outside this area. However, the application may use this rectangle when 
deciding how to display the ink according to a given view or on the printer. 

As described above in some embodiments, transforms may be combined prior to 
application to ink in a first coordinate system. The following is an example of how two 
two-dimensional mappings may be combined into a single mapping. It is appreciated that 
more mappings in N dimensions may be combined as well but are not shown for 
simplicity. 

A first transformation mapping may be given by: 
X = ax + by + c (8) 
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Y' = dx +ey + f (9) 
[98] A second transformation mapping may be given by: 

X"=gx'+hy'+i (10) 

Y"=jx'+ky'+m (11) 
[99] These two mappings of equations may be expressed as: 

X" = g(ax +by + c) + h(dx +ey +f) + i (12) 

Y" = j(ax +by + c) + k(dx +ey +f) + m (13) 
[100] Equations 12 and 13 simplify to: 

X" = (ga +hd)x +(gb+he)y + gc + hf + i (14) 

Y" = (ja + kd)x + (ib+ke)y+jc + kf + m (15) 
[101] These equations may also be represented in matrix form with equations 8 and 9 as 
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[104] Accordingly, having the coefficients for a first mapping and a second mapping may be 
combined. The resulting mapping may then be combined with other mappings or applied 
to ink to transform the ink. 

[105] A number of tags may be used to indicate the type of transform to be applied to ink. The 
following table provides a list of tags and information relating to the coefficients for the 
transform. The values having no value (or zero as a value) may have a placeholder in the 
matrix (e.g., 0) or the placeholder may be eliminated. Eliminating the placeholder further 
reduces the size of the data structure storing the ink. 



Tag 


Coefficients 


TAG_TRANSFORM_ISOTROPIC_SCALE 


Mil = M22 nonzero value; 
M12 = M21 =DX = DY = 0 


TAG_TRANSFORM_ANISOTROPIC_SCALE 


Mil, M22 arbitrary non zero 
floats; 

M12 = M21=DX = DY = 0 


TAG_TRANSFORM_ROTATE 


Storing MBE integer number, 
0 - 36000 // 1/100 of degree units 


TAG_TRANSFORM_TRANSLATE 


Ml 1=M22=M12=M21=0; 
DX, DY arbitrary 


TAG_TRANFROM_SCALE_AND_TRANSLATE 


Ml 1 , M22, DX, DY arbitrary; 
M12=0,M21=0 



[106] Of course, other tags are possible that further define additional sets of transforms. 
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Transform Index 

[107] If only one transform is to be applied to all strokes in an ink object, then a transform 
block 1104 with identifier 1103, size/count field 1105, and coefficients 1106 may be 
specified. If more than one transform is specified in an ink object, the ink object may use 
an index to relate a stroke or series of strokes back to a specific transform (for example in 
a transform block). The transform index (which may be specified with the identifier 
TAG_TIDX) assigns a transform block to a stroke. A transform index may be followed 
by an index value that specifies the entry in the transform table. All strokes in the stream 
from that point on may use the specified transform block until the next transform index is 
encountered in the stream. 

[108] In an alternate embodiment, if there is no transform index in the stream somewhere 
before the stroke, it may be assumed that this stroke should use the 0th transform block in 
the transform table. And if there is no transform table in the stream, then no transforms 
should be applied to any stroke. 

Encoding of Values 

[109] To further assist with the storage of ink, the content of an ink object may be serialized as 

described in U.S. Serial Nos. and ( BW 03797.00132, BW 03797.00131), 

which are expressly incorporated by reference. Also, the X and Y data may be 
compressed. A number of encoding strategies and compression methods may be used 
alone or in combination. 

Sizes of Tags and Numbers 

[110] At the most basic level, the ink object may be composed of numbers. Even tags may be 
considered indexes, which are just small integer numbers. In fact, most of the time these 
numbers are small enough that they could be represented by a single byte if there was a 
way of determining when a byte represented a single number and when it was just part of 
a bigger number. In some embodiments, no encoding is used. In other embodiments, it 
may be possible to take advantage of this observation by encoding numbers using a 
multi-byte encoding technique. 
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Multi-byte encoding makes it possible to represent small numbers in one byte, larger 
numbers in two bytes and very large numbers in however many bytes are necessary. This 
means that tags, which usually have a value less than 100, are stored as a single byte and 
sizes, which may be small or large, are stored in the most efficient manner. In effect, 
multi-byte encoding may be a compression technique. 

Various types of multi-byte encoding are known. An example of multi-byte encoding is 
shown and works as follows: 

a. Numbers less than 128 are encoded in one byte. 

b. The most significant bit remains in the byte clear. 

c. Multi-byte encoding interprets the most significant bit being clear to mean this 
may be the last byte in a number. 

d. Numbers larger than 128 are broken up into 7 bit segments. 

e. The 7 bit segments are then each stored in a byte. 

f . And the most significant bit in each byte except the last may be set. 
[113] In other words, the system handles information such that: 

a. Numbers less than 2^= 128 are encoded in a single byte. 

b. Numbers less than l}^ = 16384 are encoded in two bytes. 

c. Numbers less than 2^^ = are encoded in three bytes. 

d. Etc. 

[114] In general, bytes are processed until a byte with the most significant bit clear may be 
encountered. For example, the first number encountered may be the ink object identifier 
number. For version 1.0 this value may be "0" and can be encoded in a single byte. The 
next number may be the size of the stream following the size value, and for small ink 
objects as in the first example this will also be encoded in a single byte. However, if the 
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stream may be long this value can grow as large as necessary. For example, a multi-byte 
encoded number of 10 bytes can represent a 64-bit number. 

[115] This same process may be applied to "tags" and other values in the stream. In general 
since "tags" are small integer indexes, they too may be one byte encoded. 

Multi-byte Encoding of Signed Numbers 

[116] Multi-byte encoding as described above works well for positive integers. However, in 
some cases it may be necessary to store signed numbers. For example, the coordinates of 
a point may be positive or negative depending on where the application situates the 
origin. 

[117] To multi-byte encode a signed number, the absolute value of the signed number may be 
determined, the absolute value then may be shifted left by 1 bit, and the sign of the 
original number may be stored in the list significant bit. 

[118] Using the technique set forth above, the signed numbers with absolute values are handled 
as follows: 

a. Numbers less than 2^ = 64 are encoded in one byte, 

b. Numbers less than iP = 8192 are encoded in 2 bytes 

c. etc. 

[119] Further, ink may be compressed using known compression algorithms (for example, 
using Huffman compression systems and the Hke). 

A Summarization of the Storage of Ink 

[120] A transform mapping may be stored in an ink object. The transform mapping permits the 
ink object to be stored in a first set of coordinates and later transformed to a second set of 
coordinates when needed. 

[121] Although the invention has been defined using the appended claims, these claims are 
exemplary in that the invention may be intended to include the elements and steps 
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described herein in any combination or sub combination. Accordingly, there are any 
number of alternative combinations for defining the invention, which incorporate one or 
more elements from the specification, including the description, claims, and drawings, in 
various combinations or sub combinations. It will be apparent to those skilled in the 
relevant technology, in light of the present specification, that alternate combinations of 
aspects of the invention, either alone or in combination with one or more elements or 
steps defined herein, may be utilized as modifications or alterations of the invention or as 
part of the invention. It may be intended that the written description of the invention 
contained herein covers all such modifications and alterations. For instance, in various 
embodiments, a certain order to the data has been shown. However, any reordering of the 
data is encompassed by the present invention. Also, where certain units of properties such 
as size (e.g., in bytes or bits) are used, any other units are also envisioned. 
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